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© A device for omnidirectional image viewing providing pan-and-tilt orientation, rotation, and magnification 
within a selected field-of-view for use in any application including inspection, monitoring, surveillance, and target 
acquisition.. The imaging device (using optical or infrared images) is based on the effect that the image from a 
wide angle lens, which produces a circular image of an entire field-of-view, can be mathematically corrected 
using high speed electronic circuitry. More specifically, an incoming image from any image acquisition source is 
captured in memory of the device, a transformation is performed for the viewing region of interest and viewing 
direction, and a corrected image is output as a video image signal for viewing, recording, or analysis. Multiple 
simultaneous images can be output from a single input image. As a result, this device can accomplish the 
functions of pan, tilt, rotation, and zoom throughout the selected field-of-view without the need for any 
mechanical mechanisms to move a camera. The preferred embodiment of the image transformation device can 
provide corrected images at real-time rates compatible with standard video equipment. The device can be 
controlled by discrete user input, automatic computer scanning, or discrete environmental inputs to select 
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desired output images. 
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This is a Continuation-in-part patent application based upon parent application Serial Number 
07/699,366 filed on May 13, 1991, for which a Notice of Allowance has been received. 

Technical Field 

5 

The present invention relates to an apparatus, algorithm, and method for transforming single perspec- 
tive-distorted field-of-view images into multiple non-distorted, normal perspective image at any orientation, 
rotation, and magnification within the field-of-view, and for using the resultant system for surveillance 
applications. The viewing direction, orientation, and magnification are controlled by either computer or 
to remote control means. More particularly, this apparatus is the electronic equivaJent ol a mechanical pan, tilt, 
zoom, and rotation camera viewing system with no moving mechanisms, and is typically utilized for 
automatic or manual surveillance of a selected environment. 

Background Art 

15 

Camera viewing systems are utilized in abundance for surveillance, inspection, security, and remote 
sensing. Remote viewing is critical, for example, for robotic manipulation tasks. Close viewing is necessary 
for detailed manipulation tasks while wide-angle viewing aids positioning of the robotic system to avoid 
collisions with the work space. The majority of these systems use either a fixed-mount camera with a 

20 limited viewing field to reduce distortion, or they utilize mechanical pan-and-tilt platforms and mechanized 
zoom lenses to orient the camera and magnify its image. In the applications where orientation of the camera 
and magnification of its image are required, the mechanical solution is large in size and can subtend a 
significant volume making the viewing system difficult to conceal or use in close quarters. Several cameras 
are usually necessary to provide wide-angle viewing of the work space. 

25 In order to provide a maximum amount of viewing coverage or subtended angle, mechanical pan/tilt 
mechanisms usually use motorized drives and gear mechanisms to manipulate the vertical and horizontal 
orientation. An example of such a device is shown in . U.S. Patent Number 4,728,839 issued to J.B. 
Coughlan, et al, on March 1, 1988. Collisions with the working environment caused by these mechanical 
pan/tilt orientation mechanisms can damage both the camera and the work space and impede the remote 

30 handling operation. Simultaneously, viewing in said remote environments is extremely important to the 
performance of inspection and manipulation activities. 

Camera viewing systems that use internal optics to provide wide viewing angles have also been 
developed in order to minimize the size and volume of the camera and the intrusion into the viewing area. 
These systems rely on the movement of either a mirror or prism to change the tilt-angle of orientation and 

35 provide mechanical rotation of the entire camera to change the pan angle of orientation. Additional lenses 
are used to minimize distortion. Using this means, the size of the camera orientation system can be 
minimized, but "blind spots'* in the center of the view result. Also, these systems typically have no means 
of magnifying the image and or producing multiple images from a single camera. 

References that may be relevant to the evaluation of the present invention are U. S. Patent Numbers: 

40 4,772.942 issued to M. J. Tuck on September 20, 1988; 5,023,725 issued to D. McCutchen on June 11, 
1991; 5,067,019 issued to R. D. Juday on November 19, 1991; and 5,068,735 issued to K. Tuchiya, et al on 
November 26, 1991. 

Accordingly, it is an object of the present invention to provide an apparatus that can provide an image 
of any portion of the viewing space within a selected field-of-view without moving the apparatus, and then 
45 electronically correct for visual distortions of the view. 

It is another object of the present invention to provide horizontal orientation (pan), vertical orientation 
(tilt) and rotational orientation (rotation) of the viewing direction with no moving mechanisms. 

It is another object of the present invention to provide the ability to magnify or scale the image (zoom in 
and out) electronically. 

so it is another object of the present invention to provide electronic control of the image intensity (iris 
level). 

It is another object of the present invention to be able to accomplish pan, tilt, zoom, rotation, and iris 
adjustments with simple inputs made by a lay person from a joystick, keyboard controller, or computer 
controlled means. 

55 It is also an object of the present invention to provide accurate control of the absolute viewing direction 
and orientations using said input devices. 

A further object of the present invention is to provide the ability to produce multiple images with 
different orientations and magnifications simultaneously from a single input image. 
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Another object of the present invention is to be able to provide these images at real-time video rates, 
e.g. thirty transformed images per second, and to support various display format standards such as the 
National Television Standards Committee RS-170 signaJ format and/or higher resolution formats currently 
under development. 

It is also an object of the present invention to provide a system that can be used for automatic or 
manual surveillance of selected environments, with optical views of these environments corrected electron- 
ically to remove distortion so as to facilitate this surveillance. 

These and other objects of the present invention will become apparent upon consideration of the 
drawings hereinafter in combination with a complete description thereof. 

Disclosure of the Invention 
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In accordance with the present invention, there is provided an omnidirectional viewing system that 
produces the equivalent of pan, tilt, zoom, and rotation within a selected field-of-view with no moving parts. 
Further, the present invention includes means for controlling this omnidirectional viewing in surveillance 
applications. This device includes a means for digitizing an incoming or prerecorded video image signal, 
transforming a portion of the video image based upon operator or preselected commands, and producing 
one or more output images that are in correct perspective for human viewing. In one embodiment, the 
incoming image is produced by a fisheye lens which has a wide angle field-of-view. This image is captured 
into an electronic memory buffer. A portion of the captured image, either in real time or as prerecorded, 
containing a region-of-interest is transformed into a perspective correct image by an image processing 
computer. The image processing computer provides direct mapping of the image region-of-interest into a 
corrected image using an orthogonal set of transformation algorithms. The viewing orientation is designated 
by a command signal generated by either a human operator or computerized input. The transformed image 
is deposited in a second electronic memory buffer where it is then manipulated to produce the output 
image or images as requested by the command signal. This is coupled with appropriate alarms and other 
outputs to provide a complete surveillance system for selected environments. 

Brief Description of the Drawings 



Figure 1 shows a schematic block diagram of the signal processing portion of the present invention 
illustrating the major components thereof. 

Figure 2 is an exemplary drawing of a typical fisheye image used as input by the present invention. 
Lenses having other field-of-view values will produce images with similar distortion, particularly when the 
35 field-of-view is about eighty degrees or greater. 

Figure 3 is an exemplary drawing of the output image after correction for a desired image orientation 
and magnification within the original image. 

Figure 4 is a schematic diagram of the fundamental geometry that the present invention embodies to 
accomplish the image transformation. 
40 Figure 5 is a schematic diagram demonstrating the projection of the object plane and position vector 
into image plane coordinates. 

Figure 6 is a block diagram of the present invention as utilized for surveillance/inspection applications 
incorporating the basic transformation of video images obtained with, for example, wide angle lenses to 
correct for optical distortions due to the lenses, together with the control of the surveillance/inspection and 
45 appropriate alarm systems. 

Figures 7A and 7B, together, show a logic flow diagram illustrating one specific embodiment of 
controller operation for manual and automatic surveillance operations of the present invention. 



Best Mode for Carrying Out the Invention 



In order to minimize the size of the camera orientation system while maintaining the ability to zoom, a 
camera orientation system that utilizes electronic image transformations rather than mechanisms was 
developed. While numerous patents on mechanical pan-and-tilt systems have been filed, no approach using 
strictly electronic transforms and wide angle optics is known to have been successfully implemented. In 
55 addition, the electro-optical approach utilized in the present invention allows multiple images to be extracted 
from the output of a single camera. These images can be then utilized to energize appropriate alarms, for 
example, as a specific application of the basic image transformation in connection with a surveillance 
system. As utilized herein, the term "surveillance" has a wide range including, but not limited to 
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determining ingress or egress from a selected environment. Further, the term "wide angle" as used herein 
means a fieid-of-view of about eighty degrees or greater. Motivation for this device came from viewing 
system requirements in remote handling applications where the operating envelop of the equipment is a 
significant constraint to task accomplishment. 

5 The principles of the optical transform utilized in the present invention can be understood by reference 
to the system 10 of Figure 1. (This is also set forth in the afore-cited U. S. Patent Application Serial Number 
07/699,366 that is incorporated herein by reference.) Shown schematically at 11 is a wide angle, e.g., a 
fisheye, lens that provides an image of the environment with a 180 degree field-of-view. The lens is 
attached to a camera 12 which converts the optical image into an electrical signal. These signaJs are then 

w digitized electronically 13 and stored in an image buffer 14 within the present invention. An image 
processing system consisting of an X-MAP and a Y-MAP processor shown as 16 and 17, respectively, 
performs the two-dimensional transform mapping. The image transform processors are controlled by the 
microcomputer and control interface 15. The microcomputer control interface provides initialization and 
transform parameter calculation . for the system. The control interface also determines the desired trans- 

75 formation coefficients based on orientation angle, magnification, rotation, and light sensitivity input from an 
input means such as a joystick controller 22 or computer input means 23. The transformed image is filtered 
by a 2rdimensional convolution filter 18 and the output of the filtered image is stored in an output image 
buffer 19. The output image buffer 19 is scanned out by display electronics 20 to a video display device 21 
for viewing. 

20 A range of lens types can be accommodated to support various fields of view. The lens optics 11 
correspond directly with the mathematical coefficients used with the X-MAP and Y-MAP processors 16 and 
17 to transform the image. The capability to pan and tilt the output image remains even though a different 
maximum field of view is provided with a different lens element. 

The invention can be realized by proper combination of a number of optical and electronic devices. The 

25 lens 11 is exemplified by any of a series of wide angle lenses from, for example, Nikon, particularly the 
8mm F2.8. Any video source 12 and image capturing device 13 that converts the optical image into 
electronic memory can serve as the input for the invention such as a Videk Digital Camera interfaced with 
Texas Instrument's TMS 34061 integrated circuits. Input and output image buffers 14 and 19 can be 
constructed using Texas Instrument TMS44C251 video random access memory chips or their equivalents. 

30 The control interface can be accomplished with any of a number of microcontrollers including the Intel 
80C196. The X-MAP and Y-MAP transform processors 16 and 17 and image filtering 19 can be 
accomplished with application specific integrated circuits or other means as will be known to persons skilled 
in the art. The display driver can also be accomplished with integrated circuits such as the Texas 
Instruments TMS34061. The output video signal can be of the NTSC RS-170, for example, compatible with 

35 most commercial television displays in the United States. Remote control 22 and computer control 23 are 
accomplished via readily available switches and/or computer systems that also will be well known. These 
components function as a system to select a portion of the input image (fisheye or other wide angle) and 
then mathematically transform the image to provide the proper prospective for output. The keys to the 
success of the invention include: 

40 (1) the entire input image need not be transformed, only the portion of interest; 

(2) the required mathematical transform is predictable based on the lens characteristics; and 

3) calibration coefficients can be modified by the end user to correct for any lens/camera combination 

supporting both new and retrofit applications. 

The transformation that occurs between the input memory buffer 14 and the output memory buffer 19, 
45 as controlled by the two coordinated transformation circuits 16 and 17, is better understood by referring to 
Figures 2 and 3. The image shown in Figure 2 is a rendering of the image of a grid pattern produced by a 
fisheye lens. This image has a field-of-view of 180 degrees and shows the contents of the environment 
throughout an entire hemisphere. Notice that the resulting image in Figure 2 is significantly distorted relative 
to human perception. Similar distortion will be obtained even with lesser field-of-view lenses. VerticaJ grid 
50 lines in the environment appear in the image plane as 24a, 24b, and 24c. Horizontal grid lines in the 
environment appear in the image plane as 25a, 25b, and 25c. The image of an object is exemplified by 26. 
A portion of the image in Figure 2 has been corrected, magnified, and rotated to produce the image shown 
in Figure 3. Item 27 shows the corrected representation of the object in the output display. The results 
shown in the image in Figure 3 can be produced from any portion of the image of Figure 2 using the 
55 present invention. The corrected perspective of the view is demonstrated by the straightening of the grid 
pattern displayed in Figure 3. In the present invention, these transformations can be performed at real-time 
video rates (e.g., thirty times per second), compatible with commercial video standards. 
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The transformation portion of the invention as described has the capability to pan and tilt the output 
image through the entire field of view of the lens element by changing the input means, e.g. the joystick or 
computer, to the controller. This allows a large area to be scanned for information as can be useful in 
security and surveillance applications. The image can also be rotated through any portion of 360 degrees 
on .ts axis chang.ng the perceived vertical of the displayed image. This capability provides the ability to 
align the vertical .mage with the gravity vector to maintain a proper perspective in the image display 
regardless of the pan or hit angle of the image. The invention also supports modifications in the 
magnification used to display the output image. This is commensurate with a zoom function that allows a 
change in the field of view of the output image. This function is extremely useful for inspection and 
surveillance operations. The magnitude of zoom provided is a function of the resolution of the input camera 
the resolution of the output display, the clarity of the output display, and the amount of picture element 
(pixel) averaging that is used in a given display. The invention supports all of these functions to provide 
capabilities associated with traditional mechanical pan (through 180 degrees), tilt (through 180 degrees) 
rotation (through 360 degrees), and zoom devices. The digital system also supports image intensity scaling 
that emulates the functionality of a mechanical iris by shifting the intensity of the displayed image based on 
commands from the user or an external computer. 

The postulates and equations that follow are based on the image transformation portion of the present 
invention utilizing a wide angle lens as the optical element. These also apply to other field-of-view lens 
systems. There are two basic properties and two basic postulates that describe the perfect wide angle lens 
system. The first property of such a lens is that the lens has a 2* steradian field-of-view and the image it 
produces is a circle. The second property is that all objects in the field-of-view are in focus, i e the perfect 
wide angle lens has an infinite depth-of-field. The two important postulates of this lens system (refer to 
Figures 4 and 5) are stated as follows: 

Postulate 1 : Azimuth angle invariability 

- For object points that lie in a content plane that is perpendicular to the image plane and passes 
through the image plane origin, all such points are mapped as image points onto the line of 
intersection between the image plane and the content plane, i.e. along a radial line. The azimuth angle 
of the image points is therefore invariant to elevation and object distance changes within the content 
plane. 

Postulate 2: Equidistant Projection Rule 

- The radial distance, r, from the image plane origin along the azimuth angle containing the projection 
of the object point is linearly proportional to the zenith angle 0, where 0 is defined as the angle 
between a perpendicular line through the image plane origin and the line from the image plane origin 
to the object point. Thus the relationship: 

r = k/J (1) 

Using these properties and postulates as the foundation of the lens system, the mathematical 
transformation for obtaining a perspective corrected image can be determined. Figure 4 shows the 
coord.nate reference frames for the object plane and the image plane. The coordinates u.v describe object 
points within the object plane. The coordinates x.y.z describe points within the image coordinate frame of 
reference. 

The object plane shown in Figure 4 is a typical region of interest to determine the mapping relationship 
onto the image plane to properly correct the object. The direction of view vector, DOV[x,y,z] determines 
the zenith and azimuth angles for mapping the object plane, UV, onto the image plane, XY. The object 
plane is defined to be perpendicular to the vector. DOV[x.y r z]. 

The location of the origin of the object plane in terms of the image plane [x.y.z] in spherical coordinates 
is given by: 

x = D sin;? cosd 

y = D sinjff sind (2) 

z = D cos/3 

where D = scaler length from the image plane origin to the object plane origin, f3 is the zenith angle, and d 
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is the azimuth angle in image plane spherical coordinates. The origin of object plane is represented as a 
vector using the components given in Equation 1 as: 

DOV[x,y.z] = [ Dsintfcosd, Dsin0sind. Dcos/3] (3) 

5 

DOV[x,y,z] is perpendicular to the object plane and its scaler magnitude D provides the distance to the 
object plane. By aligning the YZ plane with the direction of action of DOV[x,y,z], the azimuth angle d 
becomes either 90 or 270 degrees and therefore the x component becomes zero resulting in the DOV- 
[x,y,z] coordinates: 

io 

DOV[x,y,z] = [ 0, -Dsintf, Dcos/3] (4) 

Referring now to Figure 5. the object point relative to the UV plane origin in coordinates relative to the 
origin of the image plane is given by the following: 

ts 

x = u 

y = v cos0 (5) 
z = v sin£ 

20 therefore, the coordinates of a point P(u,v) that lies in the object plane can be represented as a vector P- 
[x.y.z] in image plane coordinates: 

P[x,y,z] = [ u, vcos£, vsin/3] (6) 

25 where P[x,y,z] describes the position of the object point in image coordinates relative to the origin of the UV 
plane. The object vector 0[x,y,z] that describes the object point in image coordinates is then given by: 

0[x,y,z] = DOV[x,y,z] + P[x,y,z] (7) 

30 0[x,y,z] = [ u, vcosjS - Dsin0, vsin£ + Dco$0 (8) 

Projection onto a hemisphere of radius R attached to the image plane is determined by scaling the object 
vector 0[x,y,z] to produce a surface vector S(x,y,z]: 



40 By substituting for the components of 0[x,y.z] from Equation 8, the vector S[x,y,z] describing the image 
point mapping onto the hemisphere becomes: 

S[x y z] - RQ f u » (vcosfi-Dsinfi), (vsinfl+DcosB)] - ^ 
45 V u 2 +(vcosfl-Dsinfi) 2 +(vsinB+Dco$B) 2 

The denominator in Equation 10 represents the length or absolute value of the vector 0[x,y,z] and can 
be simplified through algebraic and trigonometric manipulation to give: 

50 

S[ x y 2 ] ROf u t (vcosfi-DsinB), (vsinfi+Dcosfi)] ^ 
V u 2 +v 2 +D 2 

55 

From Equation 1 1 , the mapping onto the two-dimensional image plane can be obtained for both x and y 

as: 
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V u 2 +v 2 +D 2 (12) 
„ R(vcosfi-DsinB) 

y " / 5 7 ==- (13) 

V u 2 +v 2 +D 2 
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Additionally, the image plane center to object plane distance D can be represented in terms of the 
image circular radius R by the relation: 

D = mR (14) 

where m represents the scale factor in radial units R from the image plane origin to the object plane 
origin. Substituting Equation 14 into Equations 12 and 13 provides a means for obtaining an effective 
scaling operation or magnification which can be used to provide zoom operation. 

Ru „^ 

* s j (15) 

V u 2 +v 2 +m 2 R 2 

25 , R(vcosB-mRsinB) 

y u 2 +v 2 +m 2 R 2 

30 Using the equations for two-dimensional rotation of axes for both the UV object plane and the XY image 
plane the last two equations can be further manipulated to provide a more general set of equations that 
provides for rotation within the image plane and rotation within the object plane. 

35 R[ uA-vB+mRsinBsin 3 ] 

X / 7 9 ~TT (17) 
y u 2 +v 2 +m 2 R 2 

40 y Rf uC-vD-mRsinBcos9 ] ^ 

V u 2 +v 2 +m 2 R 2 



45 



50 



where: 

A = (cos^cosd - sin<f>sindcos0) 
B = (sin^>cosd + cos£sindcos£ (19) 
C = (cos^sind + sin^cosdcostf) 
D = (sin^sind - cos4>cosdcos0) 



and where: 

R = radius of the image circle 

0 = zenith angle 

d = Azimuth angle in image plane 

55 * = Object plane rotation angle 

m = Magnification 

u.v = object plane coordinates 

x,y = image plane coordinates 
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The Equations 17 and 18 provide a direct mapping from the UV space to the XY image space and are 
the fundamental mathematical result that supports the functioning of the present omnidirectional viewing 
system with no moving parts. By knowing the desired zenith, azimuth, and object plane rotation angles and 
the magnification, the locations of x and y in the imaging array can be determined. This approach provides 

5 a means to transform an image from the input video buffer to the output video buffer exactly. Also, the 
image system is completely symmetrical about the zenith, therefore, the vector assignments and resulting 
signs of various components can be chosen differently depending on the desired orientation of the object 
plane with respect to the image plane. In addition, these postulates and mathematical equations can be 
modified for various lens elements as necessary for the desired field-of-view coverage in a given 

io application. 

The input means defines the zenith angle, 0, the azimuth angle, 6, the object rotation, and the 
magnification, m. These values are substituted into Equations 19 to determine values for substitution into 
Equations 17 and 18. The image circle radius, R, is a fixed value that is determined by the camera lens and 
element relationship. The variables u and v vary throughout the object plane determining the values for x 

/5 and y in the image plane coordinates. 

From the foregoing, it can be seen that a wide angle lens provides a substantially hemispherical view 
that is captured by a camera. The image is then transformed into a corrected image at a desired pan, tilt, 
magnification, rotation, and focus based on the desired view as described by a control input. The image is 
then output to a television display with the perspective corrected. Accordingly, no mechanical devices are 

20 required to attain this extensive analysis and presentation of the view of an environment through 180 
degrees of pan, 180 degrees of tilt, 360 degrees of rotation, and various degrees of zoom magnification. 

As indicated above, one application for the perspective correction of images obtained with a motionless 
wide angle camera is in the field of surveillance. The term "surveillance'' is meant to include inspection and 
like operations as well. It is often desired to continuously or periodically view a selected environment to 

25 determine activity in that environment. The term "environment" is meant to include such areas as rooms, 
warehouses, parks and the like. This activity might be, for example, ingress or egress of some object 
relative to that environment. It might also be some action that is taking place in that environment. It may be 
desired to carry out this surveillance either automatically at the desired frequency (or continuously), or upon 
demand by an operator. The size of the environment may require more than one motionless camera for 

30 complete surveillance. 

Such a surveillance system is indicated generally at 30 of Figure 6. A video camera unit 32, including a 
wide angle lens 31, is utilized to view the selected environment (or portion of the environment), with the 
output therefrom being electrical signals related to the elements as seen by the camera system. These 
signals, when present, are either directly presented to an image transformation system 33 (the components 

35 of Figure 1 without the camera/lens and the TV monitor thereof) or to a videotape recorder 34 for 
subsequent processing in the image transformation system 33. This permits evaluation during "post event" 
review as well as a review of events that occur in real time. It will be understood that additional camera-lens 
units in an environment, as well as videotape recorders, can be utilized as indicated at 35. 

Various external elements are utilized to govern the operation of the transformation system. For 

40 example, appropriate discrete switches 36 are used for the selection of the environment or portion of the 
environment to be monitored. These switches can be positioned (and operated) either at the control center 
or at the environment (or other remote location). When positioned in the environment, these switches can 
indicate some action occurring in the environment (door opening, window breaking, etc.) with the result that 
the virtual camera of the system is directed to the point of interest and then signal an external alarm for 

45 creating an audible alarm if desired. Also, alarm conditions can activate the video tape recorder discussed 
below. Since the system monitors the presence of an incoming video signal, the device can signal an alarm 
when the incoming video signal is disrupted. Where the monitoring is to be preselected, one input can be a 
computer 38. Another form of control is through the use of operator controls 40 such that the operator can 
select at any time the operation of the transformation. Options that are available in either of these types of 

so control are "Quad display" (either through the control by the computer 38 or the operator controls 40) 
wherein four displays occur on a monitor. Another option available through either control is that of 
"tweening" which is a selection of moving the effective view of the camera incrementally between active 
points or switching between active cameras within the environment. As previously described, these inputs 
are used also for selecting pan, tilt, zoom and rotation. 

55 The output of the transformation system 33 is typically in digital format As such, this output can control 
alarm enunciators 42 positioned at any location, or other forms of discrete alarms 44. They can also activate 
a videotape recording machine 46. In addition, the alarms 44 can be used to detect and announce loss of 
video signal, and permit external interrogation (manual or automated) of system status by the computer 
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interface of the system parameters including component or power failure. Such interrogation would include 
venf.cat.on of operation, video input, pan-tilt-rotation angles, magnification and setup parameters. As in the 
system of Figure 1. this surveillance system 30 provides for pictorial environment display on a TV-type 
mon.tor 48 and/or on the tape of the recording machine 46. 

Figures 7A and 7A. jointly, form a logic flow diagram that illustrates how one specific embodiment of 
the controller 30 can perform manual and automatic surveillance activities. A decision is made at 50 as to 
whether the system is under computer (external) control (see 38 of Figure 6) or manual (internal) control If 
under computer operation, the camera orientations and magnifications are communicated directly to the 
system for action at 52. In the event of internal control, it is next determined if any environmental switches 
are closed as at 54. These switches typically are hard wired, magnetic infrared or other forms that indicate 
a change in the environment in a certain location. The choice of a specific type of switch for each 
appl.cat.on will be known by persons skilled in the art. These changes (if "YES") give rise to signals at 56 
to point its visual camera in the direction of interest and then signal an external alarm for creating an 
audible alarm 42 and/or turning on the video tape recorder 34. 

After initiating these steps, or if the answer is "NO" at 54. the switches on the unit's control panel are 
read at 58 to determine the configuration and display actions needed. "Quad display" (either four displays 
or one display on the monitor 48) is checked at 60 and, if the four displays are desired (the "YES") this is 
initiated at 62. If "tweening" (incremental effective movement of a camera or'switching between cameras) is 
des.red, this is checked at 64 and the appropriate selection is made at 66. 

inputs for pan, tilt, zoom and rotation are interpreted at 68 and applied to the presently active display 
camera. Every user interaction resets the scan timer at 70 so that while the user is in control, no virtual 
camera change is occurring. When the scan time reaches zero, as monitored at 72, the next camera is 
made active and the image being displayed changes direction and/or content as at 74 to thereby update 
operation as at 52. ' K 

From the foregoing, it will be understood by those versed in the art that an advanced suiveillance 
system has been provided. This system utilizes at least one motionless video camera, having a wide angle 
ens w,th.n the environment where surveillance is desired. The perspective of images obtained with the 
lens/camera are corrected through transformation according to the technology of S. N. 07/699 366 either 
directly or after storage in a videotape recorder. Many operational conditions are selectable including tilt 
pan. zoom and rotation. Further, mufti-image displays can be obtained, and the images can be increment- 
ally scanned or switching between cameras are other options. The system provides for automatic operation 
coupled with user operation if desired. 

While certain specific elements of construction are indicated throughout the description of the present 
-nvenfion. these are given for illustration and not for limitation. Thus, the invention is to be limited only by 
trie appended claims and their equivalents. 

Claims 

1. A device for providing perspective and distortion corrected views of a selected portion of a field of view 
in a desired format, said device comprising: 

an imaging system for receiving selected optical and infrared images of said selected portion of a 
field of v.ew and for producing output signals corresponding to said selected optical and infrared 
images; 

a lens attached to said imaging system for conveyance of said selected optical and infrared images 
to said imaging system; '"ay"* 

said i rSn g a s P ^em irCUitry ** P '° V '" i ' n9 di9i,ized signals from said out P ut si 9"als from 

input image memory circuitry for receiving said digitized signals from said image capture circuitry- 
image transform circuitry for processing said digitized signals from said input image memory 

circuitry according to selected viewing pan, tilt, rotation and magnification and for producing output 

signals according to a combination of said digitized signals and said selected viewing pan, tilt rotation 

and magnification; 

output image memory circuitry for receiving said output signals from said image transform circuitry 
an input for selected user and external computer input to select said pan, tilt, rotation and 
magn.ficat.on. and for converting said selected viewing pan, tilt, rotation and magnification for input to 
said .mage transform circuitry to control said processing of said image transform circuitry and 

an output connected to said output image circuitry for recording said perspective and distortion 
corrected views according to said selected viewing pan. tilt, rotation and magnification 



10 



EP 0 610 863 A1 



2. A surveillance device for providing perspective corrected views of a selected portion of perspective 
distorted optical images from an environment into a desired format, said device comprising: 

an imaging system for receiving optical images from the environment and for producing output 
signals corresponding to said perspective distorted optical images, said imaging system including a 
5 wide angle lens for optical conveyance of said optical images to a video camera within said imaging 
system and positioned within the environment, said optical images to said video camera being 
perspective distorted by said wide angle lens; 

image capture circuitry for receiving, and producing digitized signals from, said output signals from 
said imaging system; 

to input image memory circuitry for receiving said digitized signals from said image capture circuitry; 

image transform circuitry for processing said digitized signals in said input memory circuitry 
according to selected viewing angles of pan, tilt, rotation and magnification for producing output signals 
of perspective corrected images according to a combination of said digitized signals and said selected 
viewing angles of pan. tilt, rotation and magnification; 
15 output image memory circuitry for receiving said output signals from said image transform circuitry; 

an input for selecting said viewing pan. tilt, rotation and magnification, and for producing signals to 
control said processing of said image transform circuitry as to sajd viewing pan, tilt, rotation and 
magnification; and 

an output connected to said output image circuitry for recording said perspective corrected views 
20 according to said selected viewing pan, tilt, rotation and magnification. 

3. The device of Claim 2 wherein said imaging system further comprises a video recorder for receiving 
and recording said output signals of said video camera for selective input to said image capture 
circuitry. 

25 

4. The device of Claim 2 further comprising a further input for inputting control signals to said image 
transform circuitry to achieve simultaneous multiple perspective corrected views in said output and 
selected scanning of said video camera. 

30 5. The device of Claim 4 wherein a plurality of video cameras are within the environment, each video 
camera being provided with a wide angle lens, and wherein said selected scanning of said video 
camera by said further input comprises selecting between incremental effective movement of optical 
images of a single video camera and switching between optical images of different of said plurality of 
video cameras in the environment 

35 

6. The device of Claim 4 wherein said input and said further input includes selective computer and user 
inputs. 

7. The device of Claim 4 wherein said further input includes discrete switches to control said processing 
40 of said image transform circuitry. 

8. The device of Claim 7 further comprising at least one alarm device operated by said further input 
means and by said discrete switches. 

45 9. The device of Claim 1 wherein said image transform circuitry is programmed to implement the 
following two equations: 



Rf uA-vB+mRsinflsind ] 
V u 2 +v 2 +m 2 R 2 

Rf uC-vD-mRsinflcos3 ] 
^ V u2 +v 2 + m 2 R 2 
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where: 



A = (cos^cosd - sin<f>sindcos0) 
B = (sin^cosd + cos$sindcos0) 
C = (cos^sind + sin£cosdcos0) 
D = (sin^sind - cos£cos<3cos0) 

and where: 

R = radius of the image circle 

0 - zenith angle 

b = A2imuth angle in image plane 

£ = Object plane rotation angle 

m = Magnification 
u,v = object plane coordinates 

x,y = image plane coordinates 

The device of Claim 2 wherein said image transform circuitry is programmed to implement the 
following two equations: 



Rf uA-vB+mRsinBsin3 1 
V u 2 +v 2 +m 2 R 2 



y Rf uC-vD-mRsiniteosa ) 
V u 2 +v 2 +m 2 R 2 



where: 

A = (cos^cosd - sin^sindcostf) 
B = (sin<frcosd + cos^sindcos/S) 
C = (cos^sind + sin<frcosdcosj8) 
D = (sin<frsind - cos<f>cosdcos0) 

and where: 

R = radius of the image circle 

0 = zenith angle 

d = Azimuth angle in image plane 

$ = Object plane rotation angle 

m = Magnification 

u,v = object plane coordinates 

x,y = image plane coordinates 

11. A surveillance device for providing perspective corrected views of a selected portion of perspective 
distorted optical images from an environment into a desired format, said device comprising: 

an imaging system for receiving optical images from the environment and for producing output 
signals corresponding to said perspective distorted optical images, said imaging system including a 
wide angle lens for optical conveyance of said optical images to a video camera within said imaging 
system and positioned within the environment, said optical images to said video camera being 
perspective distorted by said wide angle lens; 

image capture circuitry for receiving, and producing digitized signals from, said output signals from 
said imaging system; 

input image memory circuitry for receiving said digitized signals from said image capture circuitry; 
image transform circuitry for processing said digitized signals in said input memory circuitry 
according to selected viewing angles of pan, tilt, rotation and magnification for producing output signals 
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of perspective corrected images according to a combination of said digitized signals and said selected 
viewing angles of pan, tilt, rotation and magnification; 

output image memory circuitry for receiving said output signals from said image transform 
processor circuitry; 

5 an input for selecting said viewing pan, tilt, rotation and magnification, and for producing signals to 

control said processing of said image transform circuitry as to said viewing pan, tilt, rotation and 
magnification; and 

a further input for inputting control signals to said image transform circuitry to achieve multiple 
perspective corrected views in said output means and selected scanning of said video camera; and 
w an output connected to said output image circuitry for recording said perspective corrected views 

according to said selected viewing pan, tilt, rotation and magnification; 

wherein said image transform circuitry is programmed to implement the following two equations: 



15 R[ uA-vB+mRsi nflsin3 ] 

V u 2 +v 2 +m 2 R 2 



20 Rf uC-vD-mRsinBcos3 ] 



V u 2 +v 2 +m 2 R 2 



25 where: 

A = (cos^cosd - sin^sindcostf) 
B = (sin^cosd + cos<f>sindcos0) 
C = (cos^sind + sin4>cosdcos/9) 
30 D = (sin4>sind - cos^cosdcos/3) 

and where: 

R = radius of the image circle 
& = zenith angle 
35 6 = Azimuth angle in image plane 

4> = Object plane rotation angle 
m = Magnification 
u.v = object plane coordinates 
x.y = image plane coordinates 

40 

12. The device of Claim 11 wherein a plurality of video cameras are within the environment, each of said 
video cameras being provided with a wide angle lens, and wherein said selected scanning of said video 
cameras by said further input comprises selecting between incremental effective movement of optical 
images of a single video camera and switching between optical images of different of said plurality of 

45 video cameras in the environment. 

13. The device of Claim 11 wherein said input and said further input include selective computer and user 
inputs. 

so 14. The device of Claim 11 wherein said further input comprises discrete switches to input said selected 
pan, tilt, rotation and magnification into said image transform circuitry. 

15. The device of Claim 11 further comprising an output line from said image transform circuitry to 
selectively activate a video recorder in said image transform circuitry and an alarm. 



55 



16. The device of Claim 14 wherein said further input further comprises at least one input line for carrying 
an input signal from external sensors in the environment as to closure status of discrete switches of 
said external sensors. 
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17. The device of Claim 11 further comprising a sensor for determining existance of output signals from 
sa.d .magmg system and for energizing an alarm upon determination of absence of said output sianals 
from said imaging system. K a 

18. The device of Claim 14 wherein said further input further comprises at least one input line for carrying 
an interrogation s.gnal to said image transform circuitry to interrogate for system parameters and 
performance. 
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